feat: add streaming client-side CSV/GeoJSON/shapefile bulk export pipeline - #80
Merged
elizabetheonoja-art merged 3 commits intoJun 25, 2026
Conversation
…eline (Utility-Protocol#60) Streams filtered resource data from the REST API in chunked, gzipped NDJSON requests, transforms rows into the target format, and writes the result to disk via the File System Access API (Blob download fallback) without holding the whole dataset in memory. - types/export.ts: config, filters, progress, events and bounds (10k rows/chunk, 100-chunk cap, 50MB flush threshold, 6-decimal coord precision) - utils/ndjsonParser.ts: streaming NDJSON parser over DecompressionStream with a pako polyfill fallback - utils/formatTransformers.ts: RFC 4180 CSV, GeoJSON Point Feature (truncated coords), shapefile record extraction - utils/shapefileWriter.ts + utils/zip.ts: EPSG:4326 Point .shp/.shx/.dbf/.prj bundled into a STORED zip (with CRC32) - utils/fileWriter.ts: FS Access WritableStream writer + in-memory Blob fallback with memory-pressure warning - services/exportPipeline.ts: orchestrator with 2-chunk sliding prefetch window, memory guard, client-side filtering and progress events - hooks/useExportProgress.ts: reduces pipeline events into progress state - components/panels/ExportDialog.tsx: format selector, column picker, filter builder, progress bar + cancel - app/export: host route for the dialog - unit tests (transformers, ndjson, zip/shapefile, pipeline) + Playwright export spec verifying downloaded row count
…ds (Utility-Protocol#60) The export dialog fetches from NEXT_PUBLIC_API_URL (cross-origin from the dev server), so the Playwright route mocks now return permissive CORS headers; without them the browser blocked the responses and the schema never loaded, leaving the Export button disabled until the job timed out. Also force the Blob download fallback in the spec (headless Chromium may expose showSaveFilePicker, whose native dialog can never appear and would hang the export), and drop the forbidden Accept-Encoding header from chunk requests since the browser manages it automatically.
Contributor
|
@real-venus ci failed |
Contributor
Author
Now I am working on it. |
…tility-Protocol#60) The app's service worker intercepts fetches once it controls the page, and Playwright's page.route does not see service-worker-originated requests. The mocked /api/resources endpoints were therefore bypassed and hit the real (absent) network, failing with net::ERR_FAILED so the schema never loaded and the export hung until timeout. Blocking the service worker for this spec lets the route mocks intercept every request. Verified: full E2E suite 22/22 green.
elizabetheonoja-art
approved these changes
Jun 25, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Client-Side Bulk Export Pipeline — CSV / GeoJSON / Shapefile (#60)
Closes #60
What's included
src/types/export.tssrc/utils/ndjsonParser.tsDecompressionStream(gzip), falling back to thepakopolyfill; splits on newlines and yields parsed rows as they arrive.src/utils/formatTransformers.tsPointFeature(coordinates truncated to 6 dp), and shapefile record extraction.src/utils/shapefileWriter.ts+src/utils/zip.ts.shp/.shx/.dbf/.prjwriter, bundled into a STORED.zip(with a CRC-32 implementation).src/utils/fileWriter.tsFileWriterabstraction — streams to disk viaFileSystemWritableFileStream, or accumulates a Blob download with a memory-pressure warning when the API is unavailable.src/services/exportPipeline.tssrc/hooks/useExportProgress.tsExportProgressfor the UI.src/components/panels/ExportDialog.tsxsrc/app/export/page.tsxtests/unit/*+tests/e2e/export.spec.ts